Maximum Tests are Adaptive Permutation Tests

Invalid Permutation Tests

Permutation tests are often presented in a rather casual manner, in both introductory and advanced statistics textbooks. The appeal of the cleverness of the procedure seems to replace the need for a rigorous argument that it produces valid hypothesis tests. The consequence of this educational failing has been a widespread belief in a “permutation principle”, which is supposed invariably to give...

Permutation Tests 101

Given independent samples from P and Q, two-sample permutation tests allow one to construct exact level tests when the null hypothesis is P = Q. On the other hand, when comparing or testing particular parameters θ of P and Q, such as their means or medians, permutation tests need not be level α, or even approximately level α in large samples. Under very weak assumptions for comparing estimators...

Mantel Permutation Tests

Permutation Tests for Classification

We introduce and explore an approach to estimating statistical significance of classification accuracy, which is particularly useful in scientific applications of machine learning where high dimensionality of the data and the small number of training examples render most standard convergence bounds too loose to yield a meaningful guarantee of the generalization ability of the classifier. Instea...

Monte-Carlo Permutation Tests

The distribution of running time could be anything at all. We didnt even say whether it is continuous or discrete. Even so, the null hypothesis does induce a probability distribution on the test statistic without any other information. Specifically, it implies that the samples are indistinguishable and exchangeable. Our test statistic is a mean of 6 values (algorithm B running times) minus the ...